Fitting a Conditional Linear Gaussian Distribution

where c = (2π) is a constant and |y| = d. The j’th row of Bi is the regression vector for the j’th component of y given that Q = i. We consider tying and various constraints on the covariance matrix in order to reduce the number of free parameters. We will allow any of the variables to be hidden — we will replace observed values with expected values conditioned on evidence, as in EM. We express all the estimates in terms of expected sufficient statistics, whose size is independent of the number of samples. (This is different from the usual presentation, which give the formulas in terms of the raw data matrix.) The resulting formulas can be used in the M step of all of the following common models, which use special cases of the above equation: • Factor analysis. Q does not exist, Σ is assumed diagonal, X is hidden and Y is observed. (The temporal version of this is the Kalman filter.) • Mixture of Gaussians. X does not exist, Q is hidden, and Y is observed. (The temporal version of this is an HMM with MOG outputs.) • Mixture of factor analyzers. Σi is diagonal, Q and X are hidden, Y is observed. (The temporal version of this is a switching Kalman filter.) We assume that we have N i.i.d. training cases {et}, so the complete-data log-likelihood is